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EVALUATION  OF  SYNTHETIC  VISION  OVERLAY  CONCEPTS 
FOR  UAV  SENSOR  OPERATIONS:  LANDMARK  CUES  AND  PICTURE-IN-PICTURE 

INTRODUCTION 

Of  all  the  information  displays  in  an  Unmanned  Aerial  Vehicle  (UAV)  control  station,  video  imagery 
from  UAV-mounted  cameras  is  particularly  valuable.  Pilots  use  imagery  from  UAV  cameras  to  verify  a 
clear  path  for  ground  operations,  scan  for  air  traffic,  and  identify  navigational  landmarks  and  potential 
obstructions.  Sensor  operators  use  video  imagery  from  a  UAV  gimbal-mounted  camera  to  conduct  a 
wide  variety  of  intelligence,  surveillance  and  reconnaissance  activities  as  well  as  to  support  combat 
operations.  Video  imagery  quality,  however,  can  often  be  compromised  by  narrow  camera  field-of-view, 
datalink  degradations,  poor  environmental  conditions,  bandwidth  limitations,  or  a  highly  cluttered  visual 
scene  (e.g.,  in  urban  areas  or  mountainous  terrain). 

Synthetic  vision  technology  (also  termed  augmented  reality)  has  the  potential  to  ameliorate  negative 
video  characteristics  and  enhance  UAV  operator  interpretation  of  the  imagery.  With  this  technology, 
spatially-relevant  information,  constructed  from  databases  (e.g.,  terrain  elevation,  maps,  photo-imagery, 
etc.)  as  well  as  networked  information  sources,  can  be  represented  as  computer  symbology  and  overlaid 
conformal,  in  real  time,  onto  a  dynamic  video  image  display.  This  computer-generated  symbology 
appears  to  ‘co-exist’  with  real  objects  in  the  visual  scene,  highlighting  points  of  interest  to  operators. 

Presentation  of  a  synthetic  visual  overlay  in  a  conformal  manner  with  sensor  imagery  has  been 
demonstrated  to  reduce  scanning  time,  reduce  the  need  to  mentally  integrate  spatial  information  from 
disparate  sources,  and  facilitate  attentional  management  (Wickens  &  Long,  1995).  To  date,  research  has 
primarily  focused  on  how  synthetic  overlays  can  aid  piloting  tasks  during  manned  flights  (e.g.  Prinzel,  et 
ah,  2002).  For  instance,  flight  guidance  overlay  symbology  can  be  provided  for  reduced  visibility 
conditions,  especially  during  landings. 

It  is  anticipated  that  a  synthetic  visual  overlay  could  also  benefit  UAV  missions  (Calhoun,  Draper, 
Abernathy,  Delgado,  &  Patzek,  2005).  It  could  improve  operator  situation  awareness  by  highlighting 
elements  of  interest  within  the  video  image  such  as  threat  locations,  key  landmarks,  emergency  airfields, 
etc.  Secondly,  it  can  assist  the  operator  in  maintaining  situation  awareness  of  an  environment  if  the 
video  datalink  is  temporarily  degraded.  Synthetic  vision  overlays  could  also  serve  to  facilitate  intuitive 
communications  of  spatial  information  between  geographically  separated  users. 

It  remains  to  be  demonstrated,  however,  that  this  technology  does  indeed  benefit  UAV  operations.  It 
also  needs  to  be  determined  to  what  extent  the  overlay  increases  the  potential  for  attention/cognitive 
tunneling  to  occur  in  UAV  control  tasks.  Cognitive  tunneling  occurs  when  the  operator  becomes  fixated 
on  the  synthetic  cue  itself  (or  objects  to  which  attention  is  directed  by  the  synthetic  cue)  to  such  an  extent 
that  other,  un-cued  events  known  to  be  of  importance  are  not  attended  to  (Yeh  &  Wickens,  2001).  In  the 
case  of  UAVs,  this  may  result  in  the  failure  to  detect  unexpected,  high  priority  targets. 

Two  synthetic  vision  overlay  concepts  are  believed  to  be  particularly  valuable  for  improving  UAV 
operator  performance  in  finding  ground  targets  with  a  UAV-mounted  video  camera.  The  first  concept 
involves  the  simple  highlighting  of  key  landmarks  in  a  scene  with  computer  generated  symbology,  thus 
drawing  operator’s  attention  towards  those  areas.  The  second  is  termed  ‘picture-in-picture’  (PIP), 
whereby  synthetic-generated  imagery  surrounds  the  real  video  imagery  on  the  display.  This  concept 
affords  virtual  expansion  of  the  available  sensor  field-of-view  well  beyond  the  physical  limits  of  the 
camera,  potentially  improving  large  area  situation  awareness. 
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OBJECTIVE 


The  objective  of  this  experiment  was  to  evaluate  synthetic  visual  symbology  concepts  overlaid  on  the 
camera  display  in  a  high-fidelity  teleoperated  UAV  control  station  simulator.  Participants  performed  a 
representative  sensor  operator  task  (searching  for  ground  landmarks),  utilizing  the  following  synthetic 
overlay  concepts: 

1 .  Virtual  Flags  (white)  that  point  to  key  landmarks  within  the  scene. 


2.  Picture-in-Picture  (PIP)  whereby  the  UAV  camera  video  was  bordered  by  a  synthetically- 
generated  scene  of  the  surrounding  area,  thus  expanding  the  visible  field-of-view  virtually. 


3.  Spatial  Orientation  Symbol:  always  fixed  (not  scene-integrated)  in  bottom  right  comer  of  the 
camera  display,  indicating  current  UAV  &  sensor  orientation  as  well  as  horizontal  sensor 
coverage. 


METHOD 

Experimental  Design.  The  Spatial  Orientation  Symbol  was  present  during  all  trials.  Thus,  only 
subjective  data  was  analyzed  to  determine  if  participants  found  the  symbol  useful.  The  two  other 
overlay  concepts  were  experimentally  manipulated:  PIP  (on,  off)  and  Virtual  Flags  pointing  to 
landmarks  (on,  off).  Additionally,  the  positional  accuracy  of  the  camera’s  initial  automated  slew  to  the 
vicinity  of  the  landmark  was  examined  (low,  high).  For  some  trials  the  camera  automatically  slewed  to  a 
point  fairly  close  to  the  landmark  (i.e.,  the  landmark  was  viewable  on  the  screen  at  slew  completion). 

For  other  trials  the  camera  viewpoint  ended  up  further  away  from  the  landmark  (approximately  1.5 
screen  lengths  away  from  the  landmark)  and  the  participant  had  to  search  a  wider  area  in  order  to  find  it. 
This  factor  reflected  the  variable  accuracy  of  targeting  coordinates  in  real-world  applications.  With  two 
replications,  there  were  16  experimental  trials  to  examine  these  variables.  An  additional  8  trials  were 
included  to  explore  other  effects,  to  be  addressed  in  the  full  paper  (see  table  below). 
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Number 
of  Trials 

Trial  Type 

16 

Experimental:  PIP  (on/off),  Flags  (on/off),  Camera  Slew  Accuracy  (low/high),  &  Repetition  (2) 

2 

Unexpected  Target:  one  trial  in  Flags-On  block,  one  trial  in  Flags-Off  block  to  test  cognitive 
tunneling  effect:  are  unexpected  targets  more  likely  to  be  missed  when  flags  are  present? 

4 

*Recall:  one  trial  in  each  of  four  PIP/Flag  blocks.  Repeat  of  experimental  trial  to  test  if  PIP 
and/or  Flags  increases  situation  awareness  (SA),  speeding  up  completion  of  recall  tasks. 

2 

*Slew  error:  one  in  PIP/Flag-On  block  &  one  in  PEP/Flag-Off  block.  Camera  slewed  to 
opposite  quadrant  to  test  whether  symbology  reduces  time  to  detect  slew  error. 

*  Results  for  these  two  tasks  will  be  covered  in  full  paper. 


A  within-subjects  design  was  utilized  with  16  participants.  For  the  16  experimental  trials,  trials  with 
each  PIP  level  were  blocked,  with  participants  receiving  both  Flag  and  Slew  Accuracy  levels  (two  reps 
each)  with  one  PIP  level  prior  to  experiencing  the  other  PIP  level.  Order  of  PIP  level  was 
counterbalanced  across  participants.  The  order  of  the  Slew  Accuracy  level  was  randomized  with  the 
constraint  that  the  Low  and  High  Accuracy  trials  occurred  an  equal  number  of  times  across  the  16 
participants  in  the  first,  second,  etc.  blocks  of  experimental  trials  per  participant.  For  each  of  the  eight 
orders  of  experimental  trials,  the  recall,  unexpected  target,  and  slew  error  trials  were  next  sequentially 
inserted,  with  the  insertion  point  randomly  determined  with  the  constraint  that  none  were  the  first  trial  in 
a  block  and  the  recall  trial  was  separated  from  the  experimental  trial  recalled  by  either  one  or  two  other 
trials. 

Simulation  Environment.  The  UAV  sensor  operator  workstation  had  upper 
and  head-level  17”  color  displays,  as  well  as  two  10”  head-down  color 
displays  (HDDs).  A  map  (north-up)  was  shown  on  the  upper  display,  with 
symbology  identifying  current  UAV  location,  flight  corridor,  and  numerical 
labels  representing  landmark  locations.  The  head-level  camera  display 
presented  simulated  video  imagery  from  the  UAV’s  gimbaled  camera,  along 
with  head-up  display  sensor  symbology.  The  simulated  imagery  was  a 
realistic  database  model  of  southern  Nevada.  The  HDDs  presented  subsystem 
information.  Participants  used  the  keyboard  for  data  entry  and  manipulated 
the  right  and  left  joysticks  to  control  the  camera’s  orientation  and  zoom 
factor,  respectively. 

Tasks.  There  was  no  flight  control  task;  the  UAV  automatically  loitered  in  a  racetrack  pattern.  The 
participants’  task  was  to  manually  control  the  camera  orientation  to  locate,  zoom  in  on,  and  designate 
specific  landmarks.  Participants  completed  practice  trials  until  performance  stabilized.  For  each  trial, 
the  experimenter  issued  a  landmark  number  and  then  participants:  1)  reviewed  descriptive  information 
regarding  that  landmark,  2)  pressed  a  key  to  auto  slew  the  camera  to  landmark  area,  3)  pressed  a  key  to 
resume  manual  camera  control,  and  4)  located/designated  landmark  (centering  crosshairs  on  landmark 
and  pressing  joystick  button).  Participants  were  also  instructed:  a)  if  they  saw  a  high  priority  fuel  truck, 
they  were  to  designate  it  instead  of  the  landmark,  and  b)  if  the  camera  slew  oriented  the  view  to  the 
wrong  quadrant,  they  were  to  press  a  ‘reset’  key. 
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RESULTS 


Time  to  designate  landmarks  was  measured  as  well  as  the  number  of  unexpected  targets  detected. 
Subjective  ratings  were  obtained  via  post-trial  and  post-experiment  questionnaires.  Results  are 
presented  below  by  overlay  concept. 

Virtual  Flags;  Average  landmark  designation  time  (measured  from  manual  control  of  the  camera  to 
designation  of  the  landmark)  was  significantly  faster  when  virtual  flags  overlaid  the  imagery 
(F(l,15)=14.104,  p<0.005)  and  when  the  slew  accuracy  was  high  (F(l,15)=18.785,  p<0.005).  Moreover, 
a  significant  interaction  of  these  two  variables  (F(l,15)=7.255,  p<0.05)  suggests  that  the  flags  were 
particularly  beneficial  with  low  slew  accuracy,  decreasing  the  difference  between  low  and  high  slew 
accuracy  by  a  factor  of  2.3.  Subjective  data  indicated  that  workload  was  lower  and  that  flags  were  useful 
for  finding  landmarks  (all  p<0.01). 


Landmark  Designation  Time  with  Low 
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The  number  of  unexpected  targets  detected  with  Flags-Off  was  not  significantly  different  from  the 
number  detected  with  Flags-On  (p=0.580,  paired  samples  t-test)  indicating  that  the  overlaid  symbology 
did  not  promote  cognitive  tunneling.  However,  participants’  comments  indicated  that  the  flags  could 
pose  a  problem  in  this  way  (e.g,  .  .with  flags. .  .less  likely  to  look  for. . .”). 

Picture-in-Picture  (PIP):  It  was  hypothesized  that  the  use  of  PEP  during  low  camera  slewing  accuracy 
would  result  in  faster  landmark  designation  times  due  to  more  of  the  environment  being  displayed 
instantaneously.  However,  results  failed  to  show  any  significant  effects  (PEP  (p=0.685),  PEP  x  Slew 
Accuracy  (p=0.176),  PEP  x  Flag  (p=0.763)).  In  the  subjective  data,  participants  were  divided  on  PIP’s 
benefit.  Some  felt  that  PIP  was  distracting,  had  potential  alignment  problems,  and  was  overly  cluttered 
(the  synthetic  imagery  consisted  of  a  multi-color  aerial  map  of  the  region).  Others  felt  that  PIP  was 
useful  if  flags  were  also  available,  as  having  the  flags  in  both  views  helped  relate  the  views  and  locate 
the  landmark. 

Spatial  Orientation  Symbol:  All  16  participants  responded  that  this  symbol  was  helpful  for  locating 
targets.  However,  their  specific  comments  indicated  that  they  only  used  the  compass  points  for 
maintaining  situation  awareness,  not  the  sensor  coverage  information. 

(Significant  results  for  recall  and  slew  error  tasks  will  be  covered  in  full  paper.) 


□  Low  Slew  Accuracy 
U  High  Slew  Accuracy 


Flags-Off  Flags-On 
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CONCLUSIONS 


The  overlay  of  virtual  flags  onto  landmarks  definitely  improved  performance  and  was  favorably  rated 
by  all  participants.  The  flag  overlay  significantly  reduced  landmark  designation  time  and  served  to 
mitigate  the  negative  effect  of  low  camera  slew  accuracy,  speeding  landmark  designation  by  over  50% 
for  this  condition.  It  is  anticipated  that  the  flag  overlay  will  be  especially  beneficial  in  highly  cluttered 
imagery  or  when  landmarks  are  less  salient.  These  data  also  did  not  show  any  cognitive  tunneling 
effects  due  to  the  existence  of  virtual  flags.  Follow-on  research  is  underway  to  further  examine  this 
synthetic  vision  overlay  concept  and  to  also  test  the  implications  of  symbology  update  rate  driven  by 
UAV  positional  data. 

Contrary  to  expectations,  PIP  was  not  found  to  improve  landmark  searches.  The  most  likely 
explanation  relates  to  the  visual  clutter  in  the  synthetic  imagery,  from  incorporating  the  same  detailed 
map  that  was  used  in  the  upper  map  display.  It  was  initially  believed  that  using  the  same  map  across 
displays  would  facilitate  situation  awareness  (visual  momentum).  However,  the  degraded  resolution  and 
visual  clutter  created  by  overlaying  this  map  likely  made  perception  of  the  small  virtual  flags  difficult. 
The  type  and  amount  of  visual  information  to  display  in  the  PIP  region  will  be  addressed  in  future 
research,  as  will  other  alternative  implementations  of  this  basic  concept. 

Although  the  participants  rated  the  Spatial  Orientation  Symbol  as  useful,  it  appears  elements  other 
than  the  compass  points  were  not  beneficial  for  this  particular  task.  Future  research  will  develop  a 
tailored  display  to  better  support  large  area  searches  by  UAV  sensor  operators. 

The  results  of  the  present  study  indicate  that  the  synthetic  information  overlaid  on  the  camera  display 
has  strong  potential  for  enhancing  imagery  interpretation.  It  is  suspected  that  the  advantage  of  overlaid 
symbology  for  landmark/target  designation  and  its  ability  to  mitigate  the  effects  of  unreliable  camera 
slew  will  make  UAV  operations  more  robust  under  a  wide  variety  of  situations.  This,  in  turn,  should 
increase  UAV  mission  effectiveness  substantially. 
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Additional  Short  Abstract  Required: 

EVALUATION  OF  SYNTHETIC  VISION  OVERLAY  CONCEPTS 
FOR  UAV  SENSOR  OPERATIONS:  LANDMARK  CUES  AND  PICTURE-IN-PICTURE 

UAV  video  imagery  quality  can  be  compromised  by  narrow  field-of-view,  environmental  conditions, 
bandwidth  limitations,  or  a  highly  cluttered  scene.  Synthetic  vision  overlay  technology  (i.e.,  augmented 
reality)  can  potentially  ameliorate  video  characteristics  and  enhance  UAV  operations.  This  study 
evaluated  three  synthetic  visual  overlay  concepts  for  improving  the  situation  awareness  of  a  UAV  sensor 
operator.  Sixteen  participants  searched  for  ground  landmarks  using  a  UAV-mounted  gimbal  camera 
simulation.  Synthetic  overlay  concepts  evaluated  included  virtual  flags  on  landmarks,  a  synthetically- 
generated  imagery  border  (picture-in-picture),  and  a  display-fixed  spatial  orientation  symbol.  Results 
indicated  that  the  three  synthetic  overlay  concepts  evaluated  show  promise  for  improving  UAV 
operations.  In  particular,  virtual  flags  reduced  landmark  search  times  by  40-58%.  Although  the  picture- 
in-picture  concept  did  not  improve  search  times,  performance  and  comments  suggest  an  alternative 
instantiation  of  the  concept  to  improve  utility.  Actual  or  potential  applications  of  this  research  include 
control  of  unmanned  vehicle  systems  and  other  teleoperated  control  applications. 
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